I think multi data sources should be regarded as parallelized.

Already modified

Should we use a new key like 'sources' to describe multiple sources? The '_multiple' suffix in value seems a bit odd. Because the YAML content does not correspond one-to-one with the PipelineDef.

Already modified

sources:
 - type: mysql
    name: mysql-instance-00
    hostname: localhost
    port: 3306
    ....
 - type: mysql
    name: mysql-instance-01
    hostname: localhost
    port: 3307
    ....

And the corresponding PipelineDef looks like this:

public class PipelineDef {
    @Nullable private List<SourceDef> sources;
    private final SourceDef source;
    ...
}

If the sources is not null then we use these data sources, otherwise we use source to build up DataStream. In this way, the previous usage will not be affected.
I want to hear your opinion. @yuxiqian

I do like @ChaomingZhangCN's proposed syntax for a fully multiple data source, they're intuitive and expressive, but might be a chore if users just want to connect to a MySQL cluster with multiple servers, as they have to copy all identical configurations to both source definition.

@linjianchang's solution for now seems like MySQL specific, especially for multi-host clusters. It could not be extended for hetero-sources (like concatenating data from different DBMS), or when one wants to use different configs for each node. These cases don't exist for now since all we have is MySQL source connector, but as we're modifying composer and YAML API (instead of MySQL connector itself), such possibility should be discussed more carefully.

As for multiple sources in pipeline itself, I remembered the idea has been informally discussed with @leonardBang and @PatrickRen long time ago, and the conclusion was running multiple sources in one single job actually makes the pipeline more fragile, since any single-point failure would easily escalate and cause a global failover. Things might have changed since then, still needs hearing from senior developers on this.

sources:
 - type: mysql
    name: mysql-instance-00
    hostname: localhost
    port: 3306
    ....
 - type: mysql
    name: mysql-instance-01
    hostname: localhost
    port: 3307
    ....

And the corresponding PipelineDef looks like this:

public class PipelineDef {
    @Nullable private List<SourceDef> sources;
    private final SourceDef source;
    ...
}

If the sources is not null then we use these data sources, otherwise we use source to build up DataStream. In this way, the previous usage will not be affected. I want to hear your opinion. @yuxiqian @ChaomingZhangCN

It has been modified according to comments, please review it again, thanks!

Should be _multiple.

Already modified

@ChaomingZhangCN Hi,please help review when you have time,thanks!

It is essential to verify state compatibility.
Would adding or removing a Source from an existing job lead to state compatibility issues, or could reordering the Sources result in state inconsistency? Such incompatible operations must be exposed as errors to avoid silent handling.

It is essential to verify state compatibility. Would adding or removing a Source from an existing job lead to state compatibility issues, or could reordering the Sources result in state inconsistency? Such incompatible operations must be exposed as errors to avoid silent handling.
@lvyanquan Thanks for the review！I have already made the modifications and tested them. Please refer to the following document
https://www.yuque.com/oniucium/lfk1te/yruqcmf9vbz4rxzv?singleDoc# 《multiple source从状态恢复验证》